In a data processing system it is typically necessary to validate the output in order that errors are detected and removed. The validated output is then usable by downstream systems to achieve improved accuracy and quality of performance.
Errors can occur for a variety of reasons, such as where data becomes corrupted, data is lost due to equipment malfunction, or where data is erroneous because of errors in operation of data processing stages. In data processing systems where several dependent stages are deployed, errors quickly become compounded or exacerbated as erroneous data propagates through the data processing pipeline. Intermediate validation may occur between the different processing stages in order to try to prevent errors from propagating.
Validation itself can be a lengthy and complex process which can introduce significant latency to an overall data processing system. Where large amounts of data are to be processed existing validation processes are often unable to scale up appropriately. This is a particular problem for web-based services and/or in situations where data is to be processed in real time, for example, to control a downstream system such as a wearable computer, a robotic system or other.
The embodiments described below are not limited to implementations which solve any or all of the disadvantages of known data processing validation systems.